

<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
<head>
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  
  <title>Welcome to pspso’s documentation! &mdash; pspso 0.1.0 documentation</title>
  

  
  
  
  

  
  <script type="text/javascript" src="_static/js/modernizr.min.js"></script>
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
        <script type="text/javascript" src="_static/jquery.js"></script>
        <script type="text/javascript" src="_static/underscore.js"></script>
        <script type="text/javascript" src="_static/doctools.js"></script>
        <script type="text/javascript" src="_static/language_data.js"></script>
    
    <script type="text/javascript" src="_static/js/theme.js"></script>

    

  
  <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
  <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" /> 
</head>

<body class="wy-body-for-nav">

   
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search" >
          

          
            <a href="#" class="icon icon-home"> pspso
          

          
          </a>

          
            
            
          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <!-- Local TOC -->
              <div class="local-toc"><ul>
<li><a class="reference internal" href="#">Welcome to pspso’s documentation!</a></li>
<li><a class="reference internal" href="#overview-and-installation">Overview and Installation</a><ul>
<li><a class="reference internal" href="#overview">Overview</a></li>
<li><a class="reference internal" href="#installation">Installation</a></li>
</ul>
</li>
<li><a class="reference internal" href="#usage">Usage</a><ul>
<li><a class="reference internal" href="#mlp-example-binary-classification">MLP Example (Binary Classification)</a></li>
<li><a class="reference internal" href="#xgboost-example-binary-classification">XGBoost Example (Binary Classification)</a></li>
<li><a class="reference internal" href="#xgboost-example-regression">XGBoost Example (Regression)</a></li>
<li><a class="reference internal" href="#user-input">User Input</a></li>
</ul>
</li>
<li><a class="reference internal" href="#functions">Functions</a><ul>
<li><a class="reference internal" href="#ml-algorithms-functions">ML Algorithms Functions</a></li>
<li><a class="reference internal" href="#selection-functions">Selection Functions</a></li>
<li><a class="reference internal" href="#parameters-functions">Parameters Functions</a></li>
<li><a class="reference internal" href="#other-functions">Other Functions</a></li>
</ul>
</li>
<li><a class="reference internal" href="#module-summary">Module Summary</a></li>
<li><a class="reference internal" href="#future-work">Future Work</a><ul>
<li><a class="reference internal" href="#new-algorithms">New Algorithms</a></li>
<li><a class="reference internal" href="#cross-validation">Cross Validation</a></li>
<li><a class="reference internal" href="#multi-class-classification">Multi-Class Classification</a></li>
</ul>
</li>
<li><a class="reference internal" href="#contributing">Contributing</a></li>
<li><a class="reference internal" href="#license">License</a></li>
</ul>
</div>
            
          
        </div>
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="#">pspso</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="#">Docs</a> &raquo;</li>
        
      <li>Welcome to pspso’s documentation!</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
            
            <a href="_sources/index.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
  <img alt="alternate text" class="align-center" src="_images/LOGO.png" />
<div class="section" id="welcome-to-pspso-s-documentation">
<h1>Welcome to pspso’s documentation!<a class="headerlink" href="#welcome-to-pspso-s-documentation" title="Permalink to this headline">¶</a></h1>
</div>
<div class="section" id="overview-and-installation">
<h1>Overview and Installation<a class="headerlink" href="#overview-and-installation" title="Permalink to this headline">¶</a></h1>
<div class="section" id="overview">
<h2>Overview<a class="headerlink" href="#overview" title="Permalink to this headline">¶</a></h2>
<p><strong>pspso</strong> is a python library for selecting machine learning algorithms
parameters. The first version supports two single algorithms:
Multi-Layer Perceptron (MLP) and Support Vector Machine (SVM). It
supports two ensembles: Extreme Gradient Boosting (XGBoost) and Gradient
Boosting Decision Trees (GBDT).</p>
<p>Two types of machine learning tasks are supported by pspso:</p>
<ul class="simple">
<li><p>Regression.</p></li>
<li><p>Binary classification.</p></li>
</ul>
<p>Three scores are supported in the first version of pspso:</p>
<ul>
<li><p><strong>Regression</strong> :</p>
<blockquote>
<div><ul class="simple">
<li><p>Root Mean Square Error (RMSE)</p></li>
</ul>
</div></blockquote>
</li>
<li><p><strong>Binary Classication</strong> :</p>
<blockquote>
<div><ul class="simple">
<li><p>Area under the Curve (AUC) of the Receiver Operating Characteristic (ROC)</p></li>
<li><p>Accuracy</p></li>
</ul>
</div></blockquote>
</li>
</ul>
</div>
<div class="section" id="installation">
<h2>Installation<a class="headerlink" href="#installation" title="Permalink to this headline">¶</a></h2>
<p>Use the package manager <a class="reference external" href="https://pip.pypa.io/en/stable/">pip</a> to
install pspso.</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>pip install pspso
</pre></div>
</div>
</div>
</div>
<div class="section" id="usage">
<h1>Usage<a class="headerlink" href="#usage" title="Permalink to this headline">¶</a></h1>
<div class="section" id="mlp-example-binary-classification">
<h2>MLP Example (Binary Classification)<a class="headerlink" href="#mlp-example-binary-classification" title="Permalink to this headline">¶</a></h2>
<p><strong>pspso</strong> is used to select the machine learning algorithms parameters.
Below is an example for using the pspso to select
the parameters of the MLP. pspso handles the
MLP random weights intialization issue that may cause losing the best
solution in consecutive iterations.</p>
<p>The following example demonstrates the selection process of the MLP parameters.
A variable named <em>params</em> was not given by the user. Hence, the default search space of the MLP is loaded.
This search space contains five parameters:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;optimizer&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&quot;RMSprop&quot;</span><span class="p">,</span> <span class="s2">&quot;adam&quot;</span><span class="p">,</span> <span class="s2">&quot;sgd&quot;</span><span class="p">,</span><span class="s1">&#39;adamax&#39;</span><span class="p">,</span><span class="s1">&#39;nadam&#39;</span><span class="p">,</span><span class="s1">&#39;adadelta&#39;</span><span class="p">]</span> <span class="p">,</span>
          <span class="s2">&quot;learning_rate&quot;</span><span class="p">:</span>  <span class="p">[</span><span class="mf">0.1</span><span class="p">,</span><span class="mf">0.3</span><span class="p">,</span><span class="mi">2</span><span class="p">],</span>
          <span class="s2">&quot;neurons&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">40</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
          <span class="s2">&quot;hiddenactivation&quot;</span><span class="p">:</span> <span class="p">[</span><span class="s1">&#39;relu&#39;</span><span class="p">,</span><span class="s1">&#39;sigmoid&#39;</span><span class="p">,</span><span class="s1">&#39;tanh&#39;</span><span class="p">],</span>
          <span class="s2">&quot;activation&quot;</span><span class="p">:[</span><span class="s1">&#39;relu&#39;</span><span class="p">,</span><span class="s1">&#39;sigmoid&#39;</span><span class="p">,</span><span class="s1">&#39;tanh&#39;</span><span class="p">]}</span>
</pre></div>
</div>
<p>The task and the score were defined as <em>binary classification</em> and <em>auc</em> respectively.
Then, the PSO was used to select the parameters of the MLP.
Results are provided back to the user through the <strong>print_results()</strong> function.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">MinMaxScaler</span>
<span class="kn">from</span> <span class="nn">pspso</span> <span class="kn">import</span> <span class="n">pspso</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">datasets</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>

<span class="n">breastcancer</span> <span class="o">=</span> <span class="n">datasets</span><span class="o">.</span><span class="n">load_breast_cancer</span><span class="p">()</span>
<span class="n">data</span><span class="o">=</span><span class="n">breastcancer</span><span class="o">.</span><span class="n">data</span><span class="c1">#get the breast cancer dataset input features</span>
<span class="n">target</span><span class="o">=</span><span class="n">breastcancer</span><span class="o">.</span><span class="n">target</span><span class="c1"># target</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span><span class="n">test_size</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">,</span><span class="n">stratify</span><span class="o">=</span><span class="n">target</span><span class="p">)</span>
<span class="n">normalize</span> <span class="o">=</span> <span class="n">MinMaxScaler</span><span class="p">(</span><span class="n">feature_range</span><span class="o">=</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span><span class="c1">#normalize input features</span>
<span class="n">X_train</span><span class="o">=</span><span class="n">normalize</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
<span class="n">X_test</span><span class="o">=</span><span class="n">normalize</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_val</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_val</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span><span class="n">test_size</span><span class="o">=</span><span class="mf">0.15</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">,</span><span class="n">stratify</span><span class="o">=</span><span class="n">Y_train</span><span class="p">)</span>
<span class="n">p</span><span class="o">=</span><span class="n">pspso</span><span class="p">(</span><span class="n">estimator</span><span class="o">=</span><span class="s1">&#39;mlp&#39;</span><span class="p">,</span><span class="n">task</span><span class="o">=</span><span class="s1">&#39;binary classification&#39;</span><span class="p">,</span> <span class="n">score</span><span class="o">=</span><span class="s1">&#39;auc&#39;</span><span class="p">)</span>
<span class="n">pos</span><span class="p">,</span><span class="n">cost</span><span class="p">,</span><span class="n">duration</span><span class="p">,</span><span class="n">model</span><span class="p">,</span><span class="n">optimizer</span><span class="o">=</span><span class="n">p</span><span class="o">.</span><span class="n">fitpspso</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span><span class="n">Y_train</span><span class="p">,</span><span class="n">X_val</span><span class="p">,</span><span class="n">Y_val</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">print_results</span><span class="p">()</span><span class="c1">#print the results</span>
<span class="n">testscore</span><span class="o">=</span><span class="n">pspso</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">model</span><span class="p">,</span><span class="n">p</span><span class="o">.</span><span class="n">estimator</span><span class="p">,</span><span class="n">p</span><span class="o">.</span><span class="n">task</span><span class="p">,</span><span class="n">p</span><span class="o">.</span><span class="n">score</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_test</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="n">testscore</span><span class="p">)</span>
</pre></div>
</div>
<p>In this example, four parameters were examined: optimizer,
learning_rate, hiddenactivation, and activation. The number of neurons
in the hidden layer was kept as default.</p>
<p>Output:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">Estimator</span><span class="p">:</span> <span class="n">mlp</span>
<span class="n">Task</span><span class="p">:</span> <span class="n">binary</span> <span class="n">classification</span>
<span class="n">Selection</span> <span class="nb">type</span><span class="p">:</span> <span class="n">PSO</span>
<span class="n">Number</span> <span class="n">of</span> <span class="n">attempts</span><span class="p">:</span><span class="mi">50</span>
<span class="n">Total</span> <span class="n">number</span> <span class="n">of</span> <span class="n">combinations</span><span class="p">:</span> <span class="mi">45360</span>
<span class="n">Parameters</span><span class="p">:</span>
<span class="p">{</span><span class="s1">&#39;optimizer&#39;</span><span class="p">:</span> <span class="s1">&#39;nadam&#39;</span><span class="p">,</span> <span class="s1">&#39;learning_rate&#39;</span><span class="p">:</span> <span class="mf">0.29</span><span class="p">,</span> <span class="s1">&#39;neurons&#39;</span><span class="p">:</span> <span class="mi">4</span><span class="p">,</span> <span class="s1">&#39;hiddenactivation&#39;</span><span class="p">:</span> <span class="s1">&#39;sigmoid&#39;</span><span class="p">,</span> <span class="s1">&#39;activation&#39;</span><span class="p">:</span> <span class="s1">&#39;sigmoid&#39;</span><span class="p">}</span>
<span class="n">Global</span> <span class="n">best</span> <span class="n">position</span><span class="p">:</span> <span class="p">[</span><span class="mf">3.8997699</span>  <span class="mf">0.28725911</span> <span class="mf">4.21218138</span> <span class="mf">1.41200923</span> <span class="mf">0.84643591</span><span class="p">]</span>
<span class="n">Global</span> <span class="n">best</span> <span class="n">cost</span><span class="p">:</span> <span class="mf">0.0</span>
<span class="n">Time</span> <span class="n">taken</span> <span class="n">to</span> <span class="n">find</span> <span class="n">the</span> <span class="nb">set</span> <span class="n">of</span> <span class="n">parameters</span><span class="p">:</span> <span class="mf">160.3374378681183</span>
<span class="n">Number</span> <span class="n">of</span> <span class="n">particles</span><span class="p">:</span> <span class="mi">5</span>
<span class="n">Number</span> <span class="n">of</span> <span class="n">iterations</span><span class="p">:</span> <span class="mi">10</span>
<span class="mf">0.9867724867724867</span>
</pre></div>
</div>
</div>
<div class="section" id="xgboost-example-binary-classification">
<h2>XGBoost Example (Binary Classification)<a class="headerlink" href="#xgboost-example-binary-classification" title="Permalink to this headline">¶</a></h2>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">MinMaxScaler</span>
<span class="kn">from</span> <span class="nn">pspso</span> <span class="kn">import</span> <span class="n">pspso</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">datasets</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>

<span class="n">breastcancer</span> <span class="o">=</span> <span class="n">datasets</span><span class="o">.</span><span class="n">load_breast_cancer</span><span class="p">()</span>
<span class="n">data</span><span class="o">=</span><span class="n">breastcancer</span><span class="o">.</span><span class="n">data</span><span class="c1">#get the breast cancer dataset input features</span>
<span class="n">target</span><span class="o">=</span><span class="n">breastcancer</span><span class="o">.</span><span class="n">target</span><span class="c1"># target</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span><span class="n">test_size</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">,</span><span class="n">stratify</span><span class="o">=</span><span class="n">target</span><span class="p">)</span>
<span class="n">normalize</span> <span class="o">=</span> <span class="n">MinMaxScaler</span><span class="p">(</span><span class="n">feature_range</span><span class="o">=</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span><span class="c1">#normalize input features</span>
<span class="n">X_train</span><span class="o">=</span><span class="n">normalize</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
<span class="n">X_test</span><span class="o">=</span><span class="n">normalize</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_val</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_val</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span><span class="n">test_size</span><span class="o">=</span><span class="mf">0.15</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">,</span><span class="n">stratify</span><span class="o">=</span><span class="n">Y_train</span><span class="p">)</span>

<span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
                <span class="s2">&quot;learning_rate&quot;</span><span class="p">:</span>  <span class="p">[</span><span class="mf">0.01</span><span class="p">,</span><span class="mf">0.2</span><span class="p">,</span><span class="mi">2</span><span class="p">],</span>
                <span class="s2">&quot;max_depth&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
                <span class="s2">&quot;n_estimators&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">200</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
                <span class="s2">&quot;subsample&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mf">0.7</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">]}</span>
<span class="n">p</span><span class="o">=</span><span class="n">pspso</span><span class="p">(</span><span class="n">estimator</span><span class="o">=</span><span class="s1">&#39;xgboost&#39;</span><span class="p">,</span><span class="n">params</span><span class="o">=</span><span class="n">params</span><span class="p">,</span><span class="n">task</span><span class="o">=</span><span class="s1">&#39;binary classification&#39;</span><span class="p">,</span> <span class="n">score</span><span class="o">=</span><span class="s1">&#39;auc&#39;</span><span class="p">)</span>
<span class="n">pos</span><span class="p">,</span><span class="n">cost</span><span class="p">,</span><span class="n">duration</span><span class="p">,</span><span class="n">model</span><span class="p">,</span><span class="n">optimizer</span><span class="o">=</span><span class="n">p</span><span class="o">.</span><span class="n">fitpspso</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span><span class="n">Y_train</span><span class="p">,</span><span class="n">X_val</span><span class="p">,</span><span class="n">Y_val</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">print_results</span><span class="p">()</span><span class="c1">#print the results</span>
<span class="n">testscore</span><span class="o">=</span><span class="n">pspso</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">model</span><span class="p">,</span><span class="n">p</span><span class="o">.</span><span class="n">estimator</span><span class="p">,</span><span class="n">p</span><span class="o">.</span><span class="n">task</span><span class="p">,</span><span class="n">p</span><span class="o">.</span><span class="n">score</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_test</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="n">testscore</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="xgboost-example-regression">
<h2>XGBoost Example (Regression)<a class="headerlink" href="#xgboost-example-regression" title="Permalink to this headline">¶</a></h2>
<p>The XGBoost is an implementation of boosting decision trees.
Five parameters were utilized for selection: objective, learning rate, maximum depth, number of estimators, and subsample.
Three categorical values were selected for the objective parameter.
The learning rate parameter values range between <em>0.01</em> and <em>0.2</em> with <em>2</em> decimal point,
maximum depth ranges between <em>1</em> and <em>10</em> with <em>0</em> decimal points <em>(1,2,3,4,5,6,7,8,9,10)</em>, etc.
The task and score are selected as regression and RMSE respectively.
The number of particles and number of iterations can be left as default values if needed.
Then, a pspso instance is created. By applying the fitpspso function, the selection process is applied.
Finally, results are printed back to the user.
The best model, best parameters, score, time, and other details will be saved in the created instance for the user to check.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">MinMaxScaler</span>
<span class="kn">from</span> <span class="nn">pspso</span> <span class="kn">import</span> <span class="n">pspso</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">datasets</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>

<span class="n">boston_data</span> <span class="o">=</span> <span class="n">datasets</span><span class="o">.</span><span class="n">load_boston</span><span class="p">()</span>
<span class="n">data</span><span class="o">=</span><span class="n">boston_data</span><span class="o">.</span><span class="n">data</span>
<span class="n">target</span><span class="o">=</span><span class="n">boston_data</span><span class="o">.</span><span class="n">target</span>

<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">target</span><span class="p">,</span><span class="n">test_size</span><span class="o">=</span><span class="mf">0.1</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">normalize</span> <span class="o">=</span> <span class="n">MinMaxScaler</span><span class="p">(</span><span class="n">feature_range</span><span class="o">=</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span><span class="c1">#normalize input features</span>
<span class="n">normalizetarget</span> <span class="o">=</span> <span class="n">MinMaxScaler</span><span class="p">(</span><span class="n">feature_range</span><span class="o">=</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span><span class="c1">#normalize target</span>

<span class="n">X_train</span><span class="o">=</span><span class="n">normalize</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">X_train</span><span class="p">)</span>
<span class="n">X_test</span><span class="o">=</span><span class="n">normalize</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">Y_train</span><span class="o">=</span><span class="n">normalizetarget</span><span class="o">.</span><span class="n">fit_transform</span><span class="p">(</span><span class="n">Y_train</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>
<span class="n">Y_test</span><span class="o">=</span><span class="n">normalizetarget</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">Y_test</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">))</span>

<span class="n">X_train</span><span class="p">,</span> <span class="n">X_val</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span> <span class="n">Y_val</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">Y_train</span><span class="p">,</span><span class="n">test_size</span><span class="o">=</span><span class="mf">0.25</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">42</span><span class="p">)</span>
<span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
                <span class="s2">&quot;objective&quot;</span><span class="p">:[</span><span class="s1">&#39;reg:tweedie&#39;</span><span class="p">,</span><span class="s2">&quot;reg:linear&quot;</span><span class="p">,</span><span class="s2">&quot;reg:gamma&quot;</span><span class="p">],</span>
                <span class="s2">&quot;learning_rate&quot;</span><span class="p">:</span>  <span class="p">[</span><span class="mf">0.01</span><span class="p">,</span><span class="mf">0.2</span><span class="p">,</span><span class="mi">2</span><span class="p">],</span>
                <span class="s2">&quot;max_depth&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
                <span class="s2">&quot;n_estimators&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">200</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
                <span class="s2">&quot;subsample&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mf">0.7</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">]}</span>
<span class="n">p</span><span class="o">=</span><span class="n">pspso</span><span class="p">(</span><span class="n">estimator</span><span class="o">=</span><span class="s1">&#39;xgboost&#39;</span><span class="p">,</span><span class="n">params</span><span class="o">=</span><span class="n">params</span><span class="p">,</span><span class="n">task</span><span class="o">=</span><span class="s1">&#39;regression&#39;</span><span class="p">,</span> <span class="n">score</span><span class="o">=</span><span class="s1">&#39;rmse&#39;</span><span class="p">)</span>
<span class="n">pos</span><span class="p">,</span><span class="n">cost</span><span class="p">,</span><span class="n">duration</span><span class="p">,</span><span class="n">model</span><span class="p">,</span><span class="n">optimizer</span><span class="o">=</span><span class="n">p</span><span class="o">.</span><span class="n">fitpspso</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span><span class="n">Y_train</span><span class="p">,</span><span class="n">X_val</span><span class="p">,</span><span class="n">Y_val</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">print_results</span><span class="p">()</span><span class="c1">#print the results</span>
<span class="n">testscore</span><span class="o">=</span><span class="n">pspso</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">model</span><span class="p">,</span><span class="n">p</span><span class="o">.</span><span class="n">estimator</span><span class="p">,</span><span class="n">p</span><span class="o">.</span><span class="n">task</span><span class="p">,</span><span class="n">p</span><span class="o">.</span><span class="n">score</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">Y_test</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">testscore</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="user-input">
<h2>User Input<a class="headerlink" href="#user-input" title="Permalink to this headline">¶</a></h2>
<p>The user is required to select the type of the algorithm (‘mlp’, ‘svm’, ‘xgboost’, ‘gbdt’); the task type (‘binary classification’,’regression’), score (‘rmse’, ‘acc’, or ‘auc’). The user can keep the parameters variable empty, where a default set of parameters and ranges is loaded
for each algorithm.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pspso</span> <span class="kn">import</span> <span class="n">pspso</span>
<span class="n">task</span><span class="o">=</span><span class="s1">&#39;binary classification&#39;</span>
<span class="n">score</span><span class="o">=</span><span class="s1">&#39;auc&#39;</span>
<span class="n">p</span><span class="o">=</span><span class="n">pspso</span><span class="o">.</span><span class="n">pspso</span><span class="p">(</span><span class="s1">&#39;xgboost&#39;</span><span class="p">,</span><span class="bp">None</span><span class="p">,</span><span class="n">task</span><span class="p">,</span><span class="n">score</span><span class="p">)</span>
</pre></div>
</div>
<p>Pspso allows the user to provide a range of parameters for exploration.
The parameters vary between each algorithm. Any parameter supported by the Scikit-Learn API for GBDT and XGBoost can be added to the selection process.
A set of parameters that contains five XGBoost parameters is shown below. The parameters are encoded in JSON object that consists of <em>key,value</em> pairs:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">params</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&quot;objective&quot;</span><span class="p">:[</span><span class="s1">&#39;reg:tweedie&#39;</span><span class="p">,</span><span class="s2">&quot;reg:linear&quot;</span><span class="p">,</span><span class="s2">&quot;reg:gamma&quot;</span><span class="p">],</span>
                <span class="s2">&quot;learning_rate&quot;</span><span class="p">:</span>  <span class="p">[</span><span class="mf">0.01</span><span class="p">,</span><span class="mf">0.2</span><span class="p">,</span><span class="mi">2</span><span class="p">],</span>
                <span class="s2">&quot;max_depth&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span><span class="mi">10</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
                <span class="s2">&quot;n_estimators&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mi">2</span><span class="p">,</span><span class="mi">200</span><span class="p">,</span><span class="mi">0</span><span class="p">],</span>
                <span class="s2">&quot;subsample&quot;</span><span class="p">:</span> <span class="p">[</span><span class="mf">0.7</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">]}</span>
</pre></div>
</div>
<p>The key can be any parameter belonging to to the algorithm under investigation.
The value is a list.
Pspso will check the type of the first element in the list, which will determine if the values of the parameter are categorical or numerical.</p>
<p><strong>Categorical Parameters</strong></p>
<p>If the parameter values are <em>categorical</em>, string values are expected to be found in the list, as shown in <em>objective</em> parameter.
The values in the list will be automatically mapped into a list of integers, where each integer represents a value in the original list.
The order of the values inside the list affect the position of the value in the search space.</p>
<p><strong>Numerical Parameters</strong></p>
<p>If the parameter is numerical, a list of three elements [lb,ub, rv] is expected to be found:</p>
<ul class="simple">
<li><p><strong>lb</strong>: repesents the lowest value in the search space</p></li>
<li><p><strong>ub</strong>: represents the maximum value in the search space</p></li>
<li><p><strong>rv</strong>: represents the number of decimal points the parameter values are rounded to before being added for training the algorithm</p></li>
</ul>
<p>For e.g if you want pspso to select n_estimators, add the following list <em>[2,200,0]</em>.
By that, the lowest n_estimators will be <em>2</em>, the highest to be examined is <em>200</em>, and each possible value is rounded to an integer value ( <em>0</em> decimal points).</p>
<p><strong>Other parameters</strong></p>
<p>The user is given the chance to handle some of the default parameters
such as the number of epochs in the MLP. Although this parameter can be optimized, but its not encouraged. The user can modify this by changing a
pspso class instance. For e.g., to change the number of
epochs from default to 10 in MLP training:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pspso</span> <span class="kn">import</span> <span class="n">pspso</span>
<span class="n">task</span><span class="o">=</span><span class="s1">&#39;binary classification&#39;</span>
<span class="n">score</span><span class="o">=</span><span class="s1">&#39;auc&#39;</span>
<span class="n">p</span><span class="o">=</span><span class="n">pspso</span><span class="o">.</span><span class="n">pspso</span><span class="p">(</span><span class="s1">&#39;mlp&#39;</span><span class="p">,</span><span class="bp">None</span><span class="p">,</span><span class="n">task</span><span class="p">,</span><span class="n">score</span><span class="p">)</span><span class="c1"># in case of empty set of params (None) default search space is loaded</span>
<span class="n">p</span><span class="o">.</span><span class="n">defaultparams</span><span class="p">[</span><span class="s1">&#39;epochs&#39;</span><span class="p">]</span><span class="o">=</span><span class="mi">10</span>
</pre></div>
</div>
<p>The verbosity can be modified for any algorithm, which allows showing details of the training process:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pspso</span> <span class="kn">import</span> <span class="n">pspso</span>
<span class="n">task</span><span class="o">=</span><span class="s1">&#39;binary classification&#39;</span>
<span class="n">score</span><span class="o">=</span><span class="s1">&#39;auc&#39;</span>
<span class="n">p</span><span class="o">=</span><span class="n">pspso</span><span class="o">.</span><span class="n">pspso</span><span class="p">(</span><span class="s1">&#39;mlp&#39;</span><span class="p">,</span><span class="bp">None</span><span class="p">,</span><span class="n">task</span><span class="p">,</span><span class="n">score</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">verbosity</span><span class="o">=</span><span class="mi">1</span>
</pre></div>
</div>
<p>Early stopping rounds can alos be modified, the user can set a value different to the default value:</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">pspso</span> <span class="kn">import</span> <span class="n">pspso</span>
<span class="n">task</span><span class="o">=</span><span class="s1">&#39;binary classification&#39;</span>
<span class="n">score</span><span class="o">=</span><span class="s1">&#39;auc&#39;</span>
<span class="n">p</span><span class="o">=</span><span class="n">pspso</span><span class="o">.</span><span class="n">pspso</span><span class="p">(</span><span class="s1">&#39;xgboost&#39;</span><span class="p">,</span><span class="bp">None</span><span class="p">,</span><span class="n">task</span><span class="p">,</span><span class="n">score</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">early_stopping</span><span class="o">=</span><span class="mi">10</span>
</pre></div>
</div>
<p>Other parameters such that n_jobs in XGBoost can also be modified before the start of the selection process.</p>
</div>
</div>
<div class="section" id="functions">
<h1>Functions<a class="headerlink" href="#functions" title="Permalink to this headline">¶</a></h1>
<div class="section" id="ml-algorithms-functions">
<h2>ML Algorithms Functions<a class="headerlink" href="#ml-algorithms-functions" title="Permalink to this headline">¶</a></h2>
<table class="longtable docutils align-default">
<colgroup>
<col style="width: 10%" />
<col style="width: 90%" />
</colgroup>
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="#pspso.pspso.forward_prop_gbdt" title="pspso.pspso.forward_prop_gbdt"><code class="xref py py-obj docutils literal notranslate"><span class="pre">forward_prop_gbdt</span></code></a>(particle, task, score, …)</p></td>
<td><p>Train the GBDT after decoding the parameters in variable particle.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#pspso.pspso.forward_prop_xgboost" title="pspso.pspso.forward_prop_xgboost"><code class="xref py py-obj docutils literal notranslate"><span class="pre">forward_prop_xgboost</span></code></a>(particle, task, score, …)</p></td>
<td><p>Train the XGBoost after decoding the parameters in variable particle.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#pspso.pspso.forward_prop_svm" title="pspso.pspso.forward_prop_svm"><code class="xref py py-obj docutils literal notranslate"><span class="pre">forward_prop_svm</span></code></a>(particle, task, score, …)</p></td>
<td><p>Train the SVM after decoding the parameters in variable particle.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#pspso.pspso.forward_prop_mlp" title="pspso.pspso.forward_prop_mlp"><code class="xref py py-obj docutils literal notranslate"><span class="pre">forward_prop_mlp</span></code></a>(particle, task, score, …)</p></td>
<td><p>Train the MLP after the decoding the parameters in variable particle.</p></td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="selection-functions">
<h2>Selection Functions<a class="headerlink" href="#selection-functions" title="Permalink to this headline">¶</a></h2>
<table class="longtable docutils align-default">
<colgroup>
<col style="width: 10%" />
<col style="width: 90%" />
</colgroup>
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="#pspso.pspso.fitpspso" title="pspso.pspso.fitpspso"><code class="xref py py-obj docutils literal notranslate"><span class="pre">fitpspso</span></code></a>([X_train, Y_train, X_val, Y_val, …])</p></td>
<td><p>Select the algorithm parameters based on PSO.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#pspso.pspso.fitpsgrid" title="pspso.pspso.fitpsgrid"><code class="xref py py-obj docutils literal notranslate"><span class="pre">fitpsgrid</span></code></a>([X_train, Y_train, X_val, Y_val])</p></td>
<td><p>Select the algorithm parameters based on Grid search.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#pspso.pspso.fitpsrandom" title="pspso.pspso.fitpsrandom"><code class="xref py py-obj docutils literal notranslate"><span class="pre">fitpsrandom</span></code></a>([X_train, Y_train, X_val, …])</p></td>
<td><p>Select the algorithm parameters based on radnom search.</p></td>
</tr>
</tbody>
</table>
<p>The fitpsrandom() and fitpsgrid() were implmented as two default selection methods.
With fit random search, the number of attempts to be tried is added by the user as a variable.
In grid search, all the possible combinations are created and investigated by the package.
These functions follow the same encoding schema used in fitpspso(), and were basically added for comparison.</p>
</div>
<div class="section" id="parameters-functions">
<h2>Parameters Functions<a class="headerlink" href="#parameters-functions" title="Permalink to this headline">¶</a></h2>
<table class="longtable docutils align-default">
<colgroup>
<col style="width: 10%" />
<col style="width: 90%" />
</colgroup>
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="#pspso.pspso.read_parameters" title="pspso.pspso.read_parameters"><code class="xref py py-obj docutils literal notranslate"><span class="pre">read_parameters</span></code></a>([params, estimator, task])</p></td>
<td><p>Read the parameters provided by the user.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#pspso.pspso.decode_parameters" title="pspso.pspso.decode_parameters"><code class="xref py py-obj docutils literal notranslate"><span class="pre">decode_parameters</span></code></a>(particle)</p></td>
<td><p>Decodes the parameters of a list into a meaningful set of parameters.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#pspso.pspso.get_default_params" title="pspso.pspso.get_default_params"><code class="xref py py-obj docutils literal notranslate"><span class="pre">get_default_params</span></code></a>(estimator, task)</p></td>
<td><p>Set the default parameters of the estimator.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#pspso.pspso.get_default_search_space" title="pspso.pspso.get_default_search_space"><code class="xref py py-obj docutils literal notranslate"><span class="pre">get_default_search_space</span></code></a>(estimator, task)</p></td>
<td><p>Create a dictionary of default parameters if the user didnt provide parameters.</p></td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="other-functions">
<h2>Other Functions<a class="headerlink" href="#other-functions" title="Permalink to this headline">¶</a></h2>
<table class="longtable docutils align-default">
<colgroup>
<col style="width: 10%" />
<col style="width: 90%" />
</colgroup>
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="#pspso.pspso.f" title="pspso.pspso.f"><code class="xref py py-obj docutils literal notranslate"><span class="pre">f</span></code></a>(q, estimator, task, score, X_train, …)</p></td>
<td><p>Higher-level method to do forward_prop in the whole swarm.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#pspso.pspso.rebuildmodel" title="pspso.pspso.rebuildmodel"><code class="xref py py-obj docutils literal notranslate"><span class="pre">rebuildmodel</span></code></a>(estimator, pos, task, score, …)</p></td>
<td><p>Used to rebuild the model after selecting the parameters.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#pspso.pspso.print_results" title="pspso.pspso.print_results"><code class="xref py py-obj docutils literal notranslate"><span class="pre">print_results</span></code></a>()</p></td>
<td><p>Print the results found in the pspso instance.</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="#pspso.pspso.calculatecombinations" title="pspso.pspso.calculatecombinations"><code class="xref py py-obj docutils literal notranslate"><span class="pre">calculatecombinations</span></code></a>()</p></td>
<td><p>A function that will generate all the possible combinations in the search space.</p></td>
</tr>
<tr class="row-odd"><td><p><a class="reference internal" href="#pspso.pspso.predict" title="pspso.pspso.predict"><code class="xref py py-obj docutils literal notranslate"><span class="pre">predict</span></code></a>(model, estimator, task, score, …)</p></td>
<td><p>A function used to release the score of a model.</p></td>
</tr>
</tbody>
</table>
</div>
</div>
<div class="section" id="module-summary">
<h1>Module Summary<a class="headerlink" href="#module-summary" title="Permalink to this headline">¶</a></h1>
<dl class="class">
<dt id="pspso.pspso">
<em class="property">class </em><code class="sig-prename descclassname">pspso.</code><code class="sig-name descname">pspso</code><span class="sig-paren">(</span><em class="sig-param">estimator='xgboost'</em>, <em class="sig-param">params=None</em>, <em class="sig-param">task='regression'</em>, <em class="sig-param">score='rmse'</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso" title="Permalink to this definition">¶</a></dt>
<dd><p>This class searches for algorithm parameters by using the Particle Swarm Optimization (PSO) algorithm.</p>
<dl class="method">
<dt id="pspso.pspso.calculatecombinations">
<code class="sig-name descname">calculatecombinations</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.calculatecombinations" title="Permalink to this definition">¶</a></dt>
<dd><p>A function that will generate all the possible combinations in the search space.
Used mainly with grid search</p>
<p>Returns</p>
<dl class="simple">
<dt>combinations: list</dt><dd><p>A list that contains all the possible combinations.</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.decode_parameters">
<em class="property">static </em><code class="sig-name descname">decode_parameters</code><span class="sig-paren">(</span><em class="sig-param">particle</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.decode_parameters" title="Permalink to this definition">¶</a></dt>
<dd><p>Decodes the parameters of a list into a meaningful set of parameters.
To decode a particle, we need the following global variables:parameters,
defaultparameters, paramdetails, and rounding.</p>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.f">
<em class="property">static </em><code class="sig-name descname">f</code><span class="sig-paren">(</span><em class="sig-param">q</em>, <em class="sig-param">estimator</em>, <em class="sig-param">task</em>, <em class="sig-param">score</em>, <em class="sig-param">X_train</em>, <em class="sig-param">Y_train</em>, <em class="sig-param">X_val</em>, <em class="sig-param">Y_val</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.f" title="Permalink to this definition">¶</a></dt>
<dd><p>Higher-level method to do forward_prop in the
whole swarm.</p>
<p>Inputs</p>
<dl class="simple">
<dt>x: numpy.ndarray of shape (n_particles, dimensions)</dt><dd><p>The swarm that will perform the search</p>
</dd>
</dl>
<p>Returns</p>
<dl class="simple">
<dt>numpy.ndarray of shape (n_particles, )</dt><dd><p>The computed loss for each particle</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.fitpsgrid">
<code class="sig-name descname">fitpsgrid</code><span class="sig-paren">(</span><em class="sig-param">X_train=None</em>, <em class="sig-param">Y_train=None</em>, <em class="sig-param">X_val=None</em>, <em class="sig-param">Y_val=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.fitpsgrid" title="Permalink to this definition">¶</a></dt>
<dd><p>Select the algorithm parameters based on Grid search.</p>
<p>Grid search was implemented to match the training process with pspso and for comparison purposes.
I have to traverse each value between x_min, x_max. Create a list seperating rounding value.</p>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.fitpspso">
<code class="sig-name descname">fitpspso</code><span class="sig-paren">(</span><em class="sig-param">X_train=None</em>, <em class="sig-param">Y_train=None</em>, <em class="sig-param">X_val=None</em>, <em class="sig-param">Y_val=None</em>, <em class="sig-param">psotype='global'</em>, <em class="sig-param">number_of_particles=5</em>, <em class="sig-param">number_of_iterations=10</em>, <em class="sig-param">options={'c1': 1.49618</em>, <em class="sig-param">'c2': 1.49618</em>, <em class="sig-param">'w': 0.7298}</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.fitpspso" title="Permalink to this definition">¶</a></dt>
<dd><p>Select the algorithm parameters based on PSO.</p>
<p>Inputs</p>
<dl class="simple">
<dt>X_train: numpy.ndarray of shape (a,b)</dt><dd><p>Contains the training input features, a is the number of samples, b is the number of features</p>
</dd>
<dt>Y_train: numpy.ndarray of shape (a,1)</dt><dd><p>Contains the training target, a is the number of samples</p>
</dd>
<dt>X_train: numpy.ndarray of shape (c,b)</dt><dd><p>Contains the validation input features, c is the number of samples, b is the number of features</p>
</dd>
<dt>Y_train: numpy.ndarray of shape (c,1)</dt><dd><p>Contains the training target, c is the number of samples</p>
</dd>
<dt>number_of_particles: integer</dt><dd><p>number of particles in the PSO search space.</p>
</dd>
<dt>number_of_iterations: integer</dt><dd><p>number of iterations.</p>
</dd>
<dt>options: dictionary</dt><dd><p>A key,value dict of PSO parameters c1,c2, and w</p>
</dd>
</dl>
<p>Returns</p>
<dl class="simple">
<dt>pos: list</dt><dd><p>The encoded parameters of the best solution</p>
</dd>
<dt>cost: float</dt><dd><p>The score of the best solution</p>
</dd>
<dt>duration: float</dt><dd><p>The time taken to conduct random search.</p>
</dd>
<dt>model:</dt><dd><p>The best model generated via random search</p>
</dd>
<dt>combinations: list of lists</dt><dd><p>The combinations examined during random search</p>
</dd>
<dt>results: list</dt><dd><p>The score of each combination in combinations list</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.fitpsrandom">
<code class="sig-name descname">fitpsrandom</code><span class="sig-paren">(</span><em class="sig-param">X_train=None</em>, <em class="sig-param">Y_train=None</em>, <em class="sig-param">X_val=None</em>, <em class="sig-param">Y_val=None</em>, <em class="sig-param">number_of_attempts=20</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.fitpsrandom" title="Permalink to this definition">¶</a></dt>
<dd><p>Select the algorithm parameters based on radnom search.</p>
<p>With Random search, the process is done for number of times specified by a parameter in the function.</p>
<p>Inputs</p>
<dl class="simple">
<dt>X_train: numpy.ndarray of shape (a,b)</dt><dd><p>Contains the training input features, a is the number of samples, b is the number of features</p>
</dd>
<dt>Y_train: numpy.ndarray of shape (a,1)</dt><dd><p>Contains the training target, a is the number of samples</p>
</dd>
<dt>X_train: numpy.ndarray of shape (c,b)</dt><dd><p>Contains the validation input features, c is the number of samples, b is the number of features</p>
</dd>
<dt>Y_train: numpy.ndarray of shape (c,1)</dt><dd><p>Contains the training target, c is the number of samples</p>
</dd>
<dt>number_of_attempts: integer</dt><dd><p>The number of times random search to be tried.</p>
</dd>
</dl>
<p>Returns</p>
<dl class="simple">
<dt>pos: list</dt><dd><p>The encoded parameters of the best solution</p>
</dd>
<dt>cost: float</dt><dd><p>The score of the best solution</p>
</dd>
<dt>duration: float</dt><dd><p>The time taken to conduct random search.</p>
</dd>
<dt>model:</dt><dd><p>The best model generated via random search</p>
</dd>
<dt>combinations: list of lists</dt><dd><p>The combinations examined during random search</p>
</dd>
<dt>results: list</dt><dd><p>The score of each combination in combinations list</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.forward_prop_gbdt">
<em class="property">static </em><code class="sig-name descname">forward_prop_gbdt</code><span class="sig-paren">(</span><em class="sig-param">particle</em>, <em class="sig-param">task</em>, <em class="sig-param">score</em>, <em class="sig-param">X_train</em>, <em class="sig-param">Y_train</em>, <em class="sig-param">X_val</em>, <em class="sig-param">Y_val</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.forward_prop_gbdt" title="Permalink to this definition">¶</a></dt>
<dd><p>Train the GBDT after decoding the parameters in variable particle.
The particle is decoded into parameters of the gbdt. Then, The gbdt is trained and the score is sent back to the fitness function.</p>
<p>Inputs</p>
<dl class="simple">
<dt>particle: list of values (n dimensions)</dt><dd><p>A particle in the swarm</p>
</dd>
<dt>task: regression, binary classification</dt><dd><p>the task to be conducted</p>
</dd>
<dt>score: rmse (regression), auc (binary classification), acc (binary classification)</dt><dd><p>the type of evaluation</p>
</dd>
<dt>X_train: numpy.ndarray of shape (m, n)</dt><dd><p>Training dataset</p>
</dd>
<dt>Y_train: numpy.ndarray of shape (m,1)</dt><dd><p>Training target</p>
</dd>
<dt>X_val: numpy.ndarray of shape (x, y)</dt><dd><p>Validation dataset</p>
</dd>
<dt>Y_val: numpy.ndarray of shape (x,1)</dt><dd><p>Validation target</p>
</dd>
</dl>
<p>Returns</p>
<dl class="simple">
<dt>variable, model</dt><dd><p>the score of the trained algorithm over the validation dataset, trained model</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.forward_prop_mlp">
<em class="property">static </em><code class="sig-name descname">forward_prop_mlp</code><span class="sig-paren">(</span><em class="sig-param">particle</em>, <em class="sig-param">task</em>, <em class="sig-param">score</em>, <em class="sig-param">X_train</em>, <em class="sig-param">Y_train</em>, <em class="sig-param">X_val</em>, <em class="sig-param">Y_val</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.forward_prop_mlp" title="Permalink to this definition">¶</a></dt>
<dd><p>Train the MLP after the decoding the parameters in variable particle.</p>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.forward_prop_svm">
<em class="property">static </em><code class="sig-name descname">forward_prop_svm</code><span class="sig-paren">(</span><em class="sig-param">particle</em>, <em class="sig-param">task</em>, <em class="sig-param">score</em>, <em class="sig-param">X_train</em>, <em class="sig-param">Y_train</em>, <em class="sig-param">X_val</em>, <em class="sig-param">Y_val</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.forward_prop_svm" title="Permalink to this definition">¶</a></dt>
<dd><p>Train the SVM after decoding the parameters in variable particle.</p>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.forward_prop_xgboost">
<em class="property">static </em><code class="sig-name descname">forward_prop_xgboost</code><span class="sig-paren">(</span><em class="sig-param">particle</em>, <em class="sig-param">task</em>, <em class="sig-param">score</em>, <em class="sig-param">X_train</em>, <em class="sig-param">Y_train</em>, <em class="sig-param">X_val</em>, <em class="sig-param">Y_val</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.forward_prop_xgboost" title="Permalink to this definition">¶</a></dt>
<dd><p>Train the XGBoost after decoding the parameters in variable particle.
The particle is decoded into parameters of the XGBoost.
This function is similar to forward_prop_gbdt
The gbdt is trained and the score is sent back to the fitness function.</p>
<p>Inputs</p>
<dl class="simple">
<dt>particle: list of values (n dimensions)</dt><dd><p>A particle in the swarm</p>
</dd>
<dt>task: regression, binary classification</dt><dd><p>the task to be conducted</p>
</dd>
<dt>score: rmse (regression), auc (binary classification), acc (binary classification)</dt><dd><p>the type of evaluation</p>
</dd>
<dt>X_train: numpy.ndarray of shape (m, n)</dt><dd><p>Training dataset</p>
</dd>
<dt>Y_train: numpy.ndarray of shape (m,1)</dt><dd><p>Training target</p>
</dd>
<dt>X_val: numpy.ndarray of shape (x, y)</dt><dd><p>Validation dataset</p>
</dd>
<dt>Y_val: numpy.ndarray of shape (x,1)</dt><dd><p>Validation target</p>
</dd>
</dl>
<p>Returns</p>
<dl class="simple">
<dt>variable, model</dt><dd><p>the score of the trained algorithm over the validation dataset, trained model</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.get_default_params">
<em class="property">static </em><code class="sig-name descname">get_default_params</code><span class="sig-paren">(</span><em class="sig-param">estimator</em>, <em class="sig-param">task</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.get_default_params" title="Permalink to this definition">¶</a></dt>
<dd><p>Set the default parameters of the estimator.
This function assigns the default parameters for the user.
Each algorithm has a set of parameters. To allow the user to search for some parameters
instead of the supported parameters, this function is used to assign a default value for each parameter.
In addition, it gets other parameters for each algorithm. For e.g, it returns the number of epochs, batch_size, and loss for the mlp.</p>
<p>Inputs</p>
<dl class="simple">
<dt>estimator: string value</dt><dd><p>A string value that determines the estimator: ‘mlp’,’xgboost’,’svm’, or ‘gbdt’</p>
</dd>
<dt>task: string value</dt><dd><p>A string value that determines the task under consideration: ‘regression’ or ‘binary classification’</p>
</dd>
</dl>
<p>Returns</p>
<dl class="simple">
<dt>defaultparams: Dictionary</dt><dd><p>A dictionary that contains default parameters to be used.</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.get_default_search_space">
<em class="property">static </em><code class="sig-name descname">get_default_search_space</code><span class="sig-paren">(</span><em class="sig-param">estimator</em>, <em class="sig-param">task</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.get_default_search_space" title="Permalink to this definition">¶</a></dt>
<dd><p>Create a dictionary of default parameters if the user didnt provide parameters.</p>
<p>Inputs</p>
<dl class="simple">
<dt>estimator: string value</dt><dd><p>A string value that determines the estimator: ‘mlp’,’xgboost’,’svm’, or ‘gbdt’</p>
</dd>
<dt>task: string value</dt><dd><p>A string value that determines the task under consideration: ‘regression’ or ‘binary classification’</p>
</dd>
</dl>
<p>Returns</p>
<dl class="simple">
<dt>params: Dictionary</dt><dd><p>A dictionary that contains default parameters to be used.</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.predict">
<em class="property">static </em><code class="sig-name descname">predict</code><span class="sig-paren">(</span><em class="sig-param">model</em>, <em class="sig-param">estimator</em>, <em class="sig-param">task</em>, <em class="sig-param">score</em>, <em class="sig-param">X_val</em>, <em class="sig-param">Y_val</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.predict" title="Permalink to this definition">¶</a></dt>
<dd><p>A function used to release the score of a model.
If the score is rmse, the value is released.
If the score is acc (accuracy), 1-acc is returned back since pso applies a minimization task.
If the score is auc, 1-auc is returned back since pso applies a minization task</p>
<p>This class is static and can be used to test the model accuracy over the hold-out sample once the selection process is finalized.</p>
<p>Inputs</p>
<dl class="simple">
<dt>model:</dt><dd><p>A trained model</p>
</dd>
<dt>estimator: string value</dt><dd><p>A string value that determines the estimator: ‘mlp’,’xgboost’,’svm’, or ‘gbdt’</p>
</dd>
<dt>task: string value</dt><dd><p>A string value that determines the task under consideration: ‘regression’ or ‘binary classification’</p>
</dd>
<dt>score: string value</dt><dd><p>Determines the score (‘rmse’,’auc’,’acc’)</p>
</dd>
<dt>X_val: numpy.ndarray</dt><dd><p>Input features</p>
</dd>
<dt>Y_val: numpy.ndarray</dt><dd><p>Target</p>
</dd>
</dl>
<p>Returns</p>
<dl class="simple">
<dt>met: float</dt><dd><p>Score value of the model</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.print_results">
<code class="sig-name descname">print_results</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.print_results" title="Permalink to this definition">¶</a></dt>
<dd><p>Print the results found in the pspso instance. Expected to print general details
like estimator, task, selection type, number of attempts examined, total number of
combinations, position of the best solution, score of the best solution, parameters,
details about the pso algorithm.</p>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.read_parameters">
<em class="property">static </em><code class="sig-name descname">read_parameters</code><span class="sig-paren">(</span><em class="sig-param">params=None</em>, <em class="sig-param">estimator=None</em>, <em class="sig-param">task=None</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.read_parameters" title="Permalink to this definition">¶</a></dt>
<dd><p>Read the parameters provided by the user.</p>
<p>Inputs</p>
<dl class="simple">
<dt>params: dictionary of key,values added by the user</dt><dd><p>This dictionary determines the parameters and ranges of parameters the user wants to selection values from.</p>
</dd>
<dt>estimator: string value</dt><dd><p>A string value that determines the estimator: ‘mlp’,’xgboost’,’svm’, or ‘gbdt’</p>
</dd>
<dt>task: string value</dt><dd><p>A string value that determines the task under consideration: ‘regression’ or ‘binary classification’</p>
</dd>
</dl>
<p>Returns</p>
<dl class="simple">
<dt>parameters</dt><dd><p>The parameters selected by the user</p>
</dd>
<dt>defaultparams</dt><dd><p>Default parameters</p>
</dd>
<dt>x_min: list</dt><dd><p>The lower bounds of the parameters search space</p>
</dd>
<dt>x_max: list</dt><dd><p>The upper bounds of the parameters search space</p>
</dd>
<dt>rounding: list</dt><dd><p>The rounding value in each dimension of the search space</p>
</dd>
<dt>bounds: dict</dt><dd><p>A dictionary of the lower and upper bounds</p>
</dd>
<dt>dimensions: integer</dt><dd><p>Dimensions of the search space</p>
</dd>
<dt>params: Dict</dt><dd><p>Dict given by the author</p>
</dd>
</dl>
</dd></dl>

<dl class="method">
<dt id="pspso.pspso.rebuildmodel">
<em class="property">static </em><code class="sig-name descname">rebuildmodel</code><span class="sig-paren">(</span><em class="sig-param">estimator</em>, <em class="sig-param">pos</em>, <em class="sig-param">task</em>, <em class="sig-param">score</em>, <em class="sig-param">X_train</em>, <em class="sig-param">Y_train</em>, <em class="sig-param">X_val</em>, <em class="sig-param">Y_val</em><span class="sig-paren">)</span><a class="headerlink" href="#pspso.pspso.rebuildmodel" title="Permalink to this definition">¶</a></dt>
<dd><p>Used to rebuild the model after selecting the parameters.</p>
</dd></dl>

</dd></dl>

</div>
<div class="section" id="future-work">
<h1>Future Work<a class="headerlink" href="#future-work" title="Permalink to this headline">¶</a></h1>
<div class="section" id="new-algorithms">
<h2>New Algorithms<a class="headerlink" href="#new-algorithms" title="Permalink to this headline">¶</a></h2>
<p>Other machine learning algorithms and packages will be added such as the catboost.</p>
</div>
<div class="section" id="cross-validation">
<h2>Cross Validation<a class="headerlink" href="#cross-validation" title="Permalink to this headline">¶</a></h2>
<p>We are working towards adding the cross validation support that will take the training data and number of folds.</p>
<p>Then split the records and train each fold. The average performance of cross-validation will be retuned back to the user.</p>
</div>
<div class="section" id="multi-class-classification">
<h2>Multi-Class Classification<a class="headerlink" href="#multi-class-classification" title="Permalink to this headline">¶</a></h2>
<p>We are also working on adding multi-class classification and data oversampling techniques.</p>
</div>
</div>
<div class="section" id="contributing">
<h1>Contributing<a class="headerlink" href="#contributing" title="Permalink to this headline">¶</a></h1>
<p>Pull requests are welcome. For major changes, please open an issue first
to discuss what you would like to change.</p>
<p>Please make sure to update tests as appropriate.</p>
<p>We are working towards adding the cross validation support that will take the training data and number of folds, then split the records and train each fold. Finally, the average performance is retuned to the user.</p>
<p>We are also working on adding multi-class classification and data oversampling techniques.</p>
</div>
<div class="section" id="license">
<h1>License<a class="headerlink" href="#license" title="Permalink to this headline">¶</a></h1>
<p>Copyright (c) [2020] [Ali Haidar]</p>
<p>Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:</p>
<p>The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.</p>
<p>THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.</p>
</div>


           </div>
           
          </div>
          <footer>
  

  <hr/>

  <div role="contentinfo">
    <p>
        &copy; Copyright 2020, Ali Haidar

    </p>
  </div>
  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 

</footer>

        </div>
      </div>

    </section>

  </div>
  


  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>